System and methods for adapting lessons to student needs

ABSTRACT

In one embodiment, the invention discloses a method for adapting educational content. The method comprises generating data for each of a plurality of students, the data pertaining to an aspect of the student&#39;s interaction with an educational system; combining the generated data to form a combined data set; analyzing the combined data set to identify clusters, each representing similar students according to a mathematical model; and adapting the educational system to provide a customized learning experience for a particular student based on an identified cluster.

This application claims the benefit of priority to U.S. ProvisionalPatent Application Nos. 60/830,937 and 60/883,416, each of which ishereby incorporated by reference.

FIELD

Embodiments of the invention relate generally to the field of education,and more specifically to the field of delivering educational content,including lessons, to students using technological means.

BACKGROUND

Educational systems utilizing computers may be used to provideeducational content to learners. Such educational content may includemultimedia content rich in, e.g. sound, graphics, video, etc. to providea compelling learning experience. To ensure optimal learning, sucheducational systems may benefit from adapting the educational content tosuit the preferences, learning styles, and requirements of particularlearners.

SUMMARY

In one embodiment, the invention discloses a method for adaptingeducational content. The method comprises generating data for each of aplurality of students, the data pertaining to an aspect of the student'sinteraction with an educational system; combining the generated data toform a combined data set; analyzing the combined data set to identifyclusters, each representing similar students according to a mathematicalmodel; and adapting the educational system to provide a customizedlearning experience for a particular student based on an identifiedcluster.

In another embodiment, the invention discloses an educational system fordelivering lessons adapted in accordance with the above method.

Other aspects of the present invention will become apparent from thedetailed description below.

BRIEF DESCRIPTION OF THE DRAWINGS

While the appended claims set forth the features of the presentinvention with particularity, the invention, together with its objectsand advantages, will be more readily appreciated from the followingdetailed description, taken in conjunction with the accompanyingdrawings, wherein:

FIG. 1 shows a network environment within which embodiments of thepresent invention may be practiced, the environment including a clientsystem and a server system in accordance with one embodiment of theinvention.

FIG. 2 shows an exemplary event stream recording, in accordance with oneembodiment of the invention.

FIG. 3 shows an exemplary process for fitting a model, in accordancewith one embodiment of the invention.

FIG. 4 shows an exemplary process for the detection of clusters ofstudents through data mining, in accordance with one embodiment of theinvention.

FIG. 5 shows an exemplary process for running an A/B test, in accordancewith one embodiment of the invention.

FIG. 6 shows a high-level block diagram of hardware that may be used toimplement any of the client systems, or server systems of the presentinvention

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the invention. It will be apparent, however, to oneskilled in the art that the invention can be practiced without thesespecific details. In other instances, structures and devices are shownonly in block diagram form in order to avoid obscuring the invention.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the invention. The appearance of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments.

Although the following description contains many specifics for thepurposes of illustration, one skilled in the art will appreciate thatmany variations and/or alterations to said details are within the scopeof the present invention. Similarly, although many of the features ofthe present invention are described in terms of each other, or inconjunction with each other, one skilled in the art will appreciate thatmany of these features can be provided independently of other features.Accordingly, this description of the invention is set forth without anyloss of generality to, and without imposing limitations upon, theinvention.

Throughout this description, the present invention will be describedusing terminology of computers, personal computers, and the Internet,along with terminology related to current educational methods andsystems. However, one skilled in the art will appreciate that suchterminology is intended to be non-limiting.

Broadly, embodiments of the invention disclose an educational system andtechniques for adapting educational content within the educationalsystem to provide a customized learning experience for an individuallearner or student. In accordance with the techniques, data for a largenumber of students who have interacted with the educational system isanalyzed to identify clusters, each representing students that aresimilar according to a mathematical model. For learners within anidentified cluster, cluster-based adaptations of the educational contentmay be applied to the educational content. These adaptations occur withno or minimal work on the part of lesson authors.

Nothing in the invention prevents the system from being used in concertwith data provided by teachers and lesson authors in the form of “seeddata”. Such seed data may be used in order to allow the system to startfunctioning before sufficient data is available. This pre-populated seeddata will be quickly supplanted by real data collected from students andderived from such collected data. Thus, the clusters may be the fruit ofa combination of human knowledge in the form of expert/teacher knowledgeand machine learning.

Turning now to FIG. 1 of the drawings, there is shown an environment 100within which embodiments of the invention may be practiced. As will beseen, the environment 100 comprises a client system 102 coupled to aserver system 104 via a communications network 106. The client system102 may represent any client computing device including e.g. a personalcomputer (PC), a notebook computer, a smart phone, etc. At leastconceptually, the client system 102 may be thought of as including oneor more input/output (I/O) devices 108 coupled to a host system 110. Inaccordance with different embodiments, the I/O devices 108 may includeinput devices such as a computer mouse or pen, a touch-sensitive screen,joystick, data gloves, and a keyboard. Additionally, the input devices108 may further comprise input devices for capturing biometricinformation pertaining to a learner. The latter category of inputdevices may include a camera for facial expression detection and for eyemotion capture/detection, a voice recorder, and a heart rate monitor.The I/O devices 108 may include output devices such as a display, asound playback device, or a haptic device for outputting braille.Hardware that may be used to realize the client system 102, inaccordance with one embodiment, is illustrated in FIG. 6 of thedrawings. Although in FIG. 1 of the drawings only one client system 102is shown, it is to be understood that in practice several such clientsystems 102 may be coupled to the server system 104 via thecommunications network 106.

In a broad sense, the communications network 106 represents any networkcapable of bridging communications between the client system 102 and theserver system 104. For purposes of the present description, thecommunications network 106 is to be understood to include a Wide-AreaNetwork (WAN) in the form of the World Wide Web (WWW) or the Internet asit is commonly referred to. Given that the communications network 106,in one embodiment, includes the Internet, the server system 104, in oneembodiment, may comprise a Web server housed in a data center. In oneembodiment, the server system 104 may be implemented as a server farm orserver cluster. Such a server farm may be housed in a single data centeror in multiple data centers.

At a high-level, the server system 104 provides educational content inthe form of lessons that are executed on the client system 102.Responsive to the execution of the lessons, the client system 102records events that are sent back to the server system 104 in the formof an event stream.

Broadly, in one embodiment, the event stream comprises events selectedfrom the group consisting of student-generated events, system-generatedevents, and biometric information pertaining to a student. For example,the event stream may comprise a record of every action that astudent/learner has performed, including both high-level and low-levelactions. High-level actions may include, answering a problem correctlyor incorrectly, and making a choice when presented with multipleactions. Turning now to FIG. 2 of the drawings, there is shown anexemplary event stream recording, in accordance with one embodiment. Aswill be seen, the event stream recording comprises recorded events 200.Each recorded event 200 contains a set of named properties 210, each ofwhich has an associated value or set of values 220. In one embodiment,the named properties 210 comprise an event identifier, a studentidentifier, a lesson identifier, the date and time of the event and thetype of the event. In addition, as much information as is available willbe recorded as additional property values.

In one embodiment, such information may include the exact problempresented, the exact steps (correct or incorrect) that the student tookto answer the problem, the incorrect answer given, the response i.e. thetime it took to answer (tied or untied to whether the answer was rightor wrong), first, second, and other derivatives of the response time,whether the student asked for a hint, the exact audio the student wasgiven. At the low-level, events may include mouse or pen clicks, drags,and motion, keystrokes, voice input, and requests for help by thestudent, the length of time that the student has been using thecomputer, as well as biometric information pertaining to the student.Such biometric information may include the student's facial expression,eye movement, and physiological data e.g., heart rate, respiration,electrocardiogram (EKG), blood pressure, stress level, hydration level,and an indication of when the student last consumed a meal. In oneembodiment, the event stream may also include a history of every screen,problem, image, audio, and video element the student experienced.

In some cases, the event stream it may also include demographic datacomprising direct information about the student, the student'ssurroundings, the time of day, climate and current weather at thestudent's residence.

Advantageously, in one embodiment it is possible to dynamically changethe quantity and quality of data that is recorded on an ongoing andimmediate basis, based on a variety of factors. Such factors may includethe product being used, the lessons being used, lesson evaluationcriteria, and student profile options.

Focusing now on the server system 104, it will be seen that the serversystem 104 includes as its functional components an event streamrecorder 112, a data analyzer 114, and assessments engine 116, a lessonadaptor 118, and a lesson delivery engine 120. Additionally, the serversystem 104 comprises several databases including an event streamdatabase 122, a student profile database 124, a standards database 126,and a lesson database 128. Hardware that may be used to realize orimplement the server system 104, in accordance with one embodiment ofthe invention, is illustrated in FIG. 6 of the drawings. In oneembodiment, the server system 104 may be housed in a data center and mayserve a large number of students.

In one embodiment, the student profile database 124 comprises individualstudent profiles. By way of example, each student profile may comprisepersonal information such as a student's age, gender, geographiclocation, interests and hobbies, etc. Learning objectives or goals inthe form of minimum standards set by a standards authority may beresident in the standards database 126, in accordance with oneembodiment. Examples of the standards authority include State and localeducational departments. The system 104 may also include lessons storedin the lesson database 128. As will be described in greater detailbelow, in one embodiment lessons in the lesson database 128 arecustomized or adapted by the lesson adapter 118 based on clusters ofsimilar learners known to the system 104.

In one embodiment, the server system 104 receives event streams from anumber of client systems 102. The event streams may be compiled into asingle database for data analysis and data mining, allowing the systemto identify both obvious and subtle similarities and differences betweenindividual students and thereby to offer many different learningexperiences, each customized or adapted to a particular student.

In some cases, the server system 104 may be pre-populated with “seeddata” based on input from subject matter experts, but the data miningrapidly converges on data that accurately represents the experiences ofreal students. In such cases the clusters may be viewed as the fruit ofa combination of human knowledge in the form of expert/teacher knowledgeand machine learning.

As a result, the system 104 uniquely allows researchers to conductexperiments on historical data, to construct and offer experiments forcurrent students in real time, to conduct such experiments withoutadversely affecting the quality of the educational materials deliveredto the students, and to combine the analysis of such new experimentswith historical data.

In describing the operations of the server system 104 in accordance withthe invention, examples are provided in English, and specific domains ofknowledge referred to. However, it is to be appreciated that theinventive techniques and systems of the present invention areindependent of particular domains of knowledge and particular languages.Thus, for example, the techniques and systems disclosed herein may beused to provide educational content to learners in the form of athree-year-old learning her letters, a sixteen-year-old preparing forthe SAT test, a twenty-five year old preparing to take the WashingtonState Bar Examination, or a seventy-year-old preparing for a driver'slicense examination. In the above examples, the learning may be in anylanguage. Moreover, the learners themselves may be in any country.

In some embodiments, the system 104 may be implemented as a single dataserver that can serve multiple countries, languages, and domains ofknowledge.

In use, the event stream recorder 112 generates multiple event streamsand, in one embodiment, combines multiple event streams into a combineddata set to be stored in the event stream database 122. Each of theevent streams comprises data pertaining to aspects of a particularstudent's interaction with the system 106. The data is recorded by eachof the client systems 104 and transmitted to the event stream recorder112.

Once the event streams have been combined as described above, the dataanalyzer 114 analyzes the combined data set to identify clusters, eachrepresenting similar students according to a mathematical model. In oneembodiment, analyzing the combined data set to identify clusterscomprises fitting a model (sometimes referred to as a curve or formula),such as a linear model or Bayes classifier, to the combined data set.One method of fitting such a model involves successive approximation ofterms and coefficients by checking the model against the data andadjusting it, essentially training the model to match the data. FIG. 3of the drawings shows an exemplary process for fitting such a model, inaccordance with one embodiment. Once a model is trained, it may be usedto predict future data, a process similar to pattern recognition. In oneembodiment, model training is not a discrete event and can happencontinuously as new data becomes available. In addition, model trainingcan happen continuously without new data in an attempt to improve themodel. The model is, in essence, a series of constantly adjusted andimproved models over time, rather than a single, fixed model.

FIG. 3 shows an exemplary process for fitting such a model, inaccordance with one embodiment of the invention. In step 310, a formulais created for calculating success of a student at a lesson withvariable coefficients wherein each variable is a measurable data pointfrom a lesson. In step 320, initial coefficients are chosen and theformula is evaluated with the initial coefficients. These coefficientsmay be chosen randomly, but picking “good guesses” will help the systemfit the model faster. In step 330, the correlation between the model andknown success data is measured. In step 340, a number of substeps arerepeated for each of the coefficients. In step 342, alternatecoefficients for each coefficient (the “current coefficient”) arechosen. One of the alternative coefficients is higher while another islower than the current coefficient. The formula is then evaluated usingthe alternative coefficients. In step 344, the correlation between themodel and known success data is measured for each of the alternatecoefficients and these correlations are saved along with the alternatecoefficients. In step 346, the subprocess ends and the loop moves to thenext coefficient. When all coefficients have been processed, in step350, the alternate coefficient with the highest correlation is selected.In step 360, the correlation of the selected alternate coefficient iscompared with the original correlation calculated in step 330. If thecorrelation is higher, the formula is modified at step 370 to use thealternate coefficients and the process returns to step 330. Otherwise,the process ends in step 380 with the current formula being chosen asthe best fit formula.

Note that this process is for fitting a single model to all data. Inmost embodiments, multiple models will be fitted to the data, but thebasic process is similar to that described in FIG. 3.

Advantageously, there is not a single type of model in use, nor is itpossible to choose a single model type without regard to the actual databeing mined. Instead, embodiments of the invention utilize a variety ofmodeling techniques in an attempt to find the one which fits best, andadditional modeling techniques can be substituted in at any time. Insome embodiments different models are used in different situations basedon a situation-dependent differentiation criterion. Examples of thesituation-dependent differentiation criteria include subject matter andstudent-specific information such as gender, school district, andgeographic location. Given a model that is appropriate for a particularsituation, it is possible to identify clusters of students that meetparticular patterns. Some of these patterns might indicate what arereferred to as learning styles, but they may indicate other reasons forwhy students are similar. Students will get grouped based on thesimilarity of their outcomes, but they may also get grouped for otherreasons. The system 104 need not have names for these clusters, nor isit necessary to ask students or anybody else about whether a givenstudent is in the cluster or not. Given the clusters, the student'sbehavior or learning outcome in a given situation can be predicted basedon the behavior of other students in the cluster. Thus, it is notnecessary to actually map the clusters to particular learning conditionsor styles. It is only necessary to identify clusters or groups ofstudents who are similar to a particular student so thatcluster-specific lesson adaptations may be generated by the lessonadapter 118 before delivery of the adaptations by the lesson deliveryengine 120 a client system 104 for the particular student.

In one embodiment, cluster detection involves partitioning the combineddata set into subsets or clusters so that the students in each subsetshare a common trait or traits. In some cases, the process of clusterdetection may involve the use of the entire combined data set, whereasin other cases only a subset or representative sample of the data setmay be used. In such a case, actual clusters are not created. Instead,whenever it is desired to know which cluster or clusters a student isin, the formulas and conditions used to determine the clusters in thesubset or representative sample are used dynamically to determine whichclusters a student is in.

In one embodiment, the subsets or clusters are not mutually exclusive,as a student might be in multiple subsets or clusters. In some cases,the cluster detection process may be modified so that the clusters aremutually exclusive. In one embodiment, a technique known as “fuzzyclustering” may be used. With fuzzy clustering each student's degree ofmembership in a cluster is calculated as a percentage or number, such as0-100.

In accordance with embodiments of the present invention, the clustersmay be detected using different clustering techniques. Each of theclustering techniques relies on at least one method for measuring thedistance or inverse correlation between two students, for example, amodified Mahalanobis distance using selected student profile values,lesson and problem result values, and event stream data for thestudents. For example, a distance measurement might be based on successwith a particular problem type, success at a particular lesson and lackof success at a particular problem type within the lesson, or firstderivative of response speed for successfully completed problems of aparticular type. Note that there is not a single distance measurementused in clustering—rather, the system will select multiple, differentdistance measurements over time in order to gain as much information aspossible about the natural clusters of students. As in the process offitting a model to the data, the system can iterate until it finds thebest distance measures that provide meaningful clusters.

FIG. 4 shows such an exemplary process 400 for the detection of clustersof students through data mining, in accordance with one embodiment ofthe invention. Referring to FIG. 4, in step 410, a number of substeps isrepeated for each of the distance measuring techniques chosen to betested. In step 415, a random but representative subset of the studentdata is selected. In step 420, a distance matrix of the subset studentdata, in which the distance value from every student to every otherstudent within the subset student data, is created. In step 425, astudent that has not been put into a cluster is selected. In step 430,the closest already-existing cluster to the selected student is found,using an algorithm such as average distance calculation, in which thecluster's value for any value being tested by the distance measurementtechnique is assumed to be the average of all such values for allstudents in the cluster. In step 435, the closest non-clustered studentto the selected student is found. In step 440, the distances to thealready-existing cluster and the non-clustered student are compared. Ifthe already-existing cluster is closer to the selected student, in step445, the selected student is added to that cluster. Otherwise, in step450, the selected student and the non-clustered student are combinedinto a new cluster. In either case, in step 455, the subset student datais examined to see if all students have been placed in a cluster. Ifnot, the process returns to step 425. Otherwise, in step 460, thesubprocess ends and the loop continues with the next distance measuringtechnique being tested. When all distance measuring techniques have beentested, in step 470, the process ends with a set of clusters determinedaccording to the distance measurement techniques used.

Note that this process performs clustering on a single distancetechnique at a time and does not iterate after all students have beenplaced into clusters. In most embodiments, multiple distance techniqueswill be used at the same time and the process will recurse as necessaryto promote clusters. In the case of fuzzy clustering, a student's degreeof measurement in a cluster might be calculated by measuring thedistance of the student from the cluster, using an algorithm such asthat described in the description of step 430 above. But, in any case,the basic process is similar to that described in FIG. 4.

The correctness or otherwise of particular student responses isdetermined, in one embodiment, by the assessments engine 116. Theassessments engine 116 may also determine whether a particular studenthas met the standards defined in the standards database 126 for thestudent.

In some embodiments, in addition to adapting a lesson based on alearner's membership in a particular cluster, the lesson adapter 118 mayalso use information from the learner's own personal profile, includingwhatever has been learned about the learner's relative success withdifferent learning styles or methods of teaching.

As an example, in early mathematics, it is known that most children havean easier time learning the concept of more than the concept of less.Similarly, most children have an easier time learning addition than theydo learning subtraction. In traditional teaching and in educationalsoftware known to the inventors, this fact that applies to most childrengets applied to all children, so more is taught before less and additionis taught before subtraction. This works for most students because theycan leverage what they find easier to learn what they find harder. Inaccordance with the above-described techniques of the present invention,a cluster of students would be detected with similar outcomes in theselessons. The detection of this unnamed cluster of students makes itpossible to apply adaptations as will be discussed later.

As noted elsewhere, there is not a single sequence of lessons for allstudents and students usually have some control over which lesson theydo at any given time. This means that different students may get lessonsin different orders, in what may be referred to as micro-sequences oflessons. The micro-sequences may be of any length and some suchmicro-sequences may be more effective or less effective than others, andsome may be more effective or less effective for certain clusters ofstudents.

The data mining described above can also identify such micro-sequencesof lessons, rate them in their effectiveness, and relate them toclusters of students. As a result, the probability that a lesson will bemade available to a student can be altered based on its perceivedeffectiveness in a given situation, possibly for a given cluster ofstudents.

Again, this information can be combined with information from thestudent's own personal data, including whatever has been learned aboutthe student's relative success with different learning styles or methodsof teaching. For example, a given student might be more successful withcertain such micro-sequences than other students. Of course, thisstudent would naturally cluster with other similar students, but thestudent's own relative success, relative to the cluster's success, couldbe used to determine which lessons are most likely to be successful.

In summary, the system 104 can recognize that adaptation can or shouldoccur for all students, for all students in a particular cluster ofstudents with similar experiences, or for a particular student, based ontheir personal profile.

Inter-Lesson Adaptation

As a result of the data mining discussed above, the system 104 candetermine that a given student is are more or less likely to besuccessful with certain lessons at a particular point in time. As aresult, the system 104 could adapt in a number of ways, including, butnot limited, to, altering a specified sequence of lessons to change theorder of the lessons, switching to a different sequence of lessons,splitting lessons into multiple lessons, combine multiple lessons into asingle lesson, and skipping lessons.

Continuous Adaptation

In other systems known to the inventors, adaptations are made inresponse to errors, or adaptations are made to reflect a known learningstyle of the student. In contrast to these inflexible approaches,embodiments of the present invention treat adaptation as a continuum andthere is continuous adapting and re-adapting to the needs of thestudent, even between problems. Thus, in the system 104 of the presentinvention adaptation is the norm, not an exception. Moreover, inaccordance with the adaptation techniques described herein there are nopreconceived notions of learning styles with the result that clustersare more flexible and dynamic.

As described above, at any given time, there is a large amount ofinformation available about each student, what the student is currentlydoing, the student's history or profile, the history of other studentsthe student is similar to, and the body of students as a whole. Thisinformation is mined to predict how the student will respond. Duringeach lesson every response by the student is noted and added to the bodyof knowledge relevant to the student in the lesson. Since the datachanges during a lesson or throughout the course of multiple lessons,the prediction can change as well. Thus, at the beginning of a lesson,the lesson might provide extra hinting or assistance and this hinting orassistance might be removed as the student learns. Similarly, the mix oftypes of problems might be changed during the lesson to give greaterconcentration to the problem types that the students need more work onor is less comfortable with. Using this method, these adaptations becomeas natural to the student as the assistance that might be given by askilled teacher.

Each component in the system may provide one or more adaptation axesalong which adaptation can occur. At one end of such an adaptation, noaccommodation is made; at the other end, a great deal of accommodationis made. For example, a component for teaching multiplication mightprovide no hint at one extreme and a 3-dimensional visual representationof the multiplication operation at the other extreme. Or a componentwhich simulates a physical object such as an abacus might provide agraphical representation of a desired state in order to aid the studentin using the object to reach that state. Each such adaptation axis isdefined in the context of the component in question and the adaptationsthat are provided are those which will assist a student in learning thesubject in question.

In addition, the system itself can adapt a lesson in a number of ways,including, but not limited to, changing the length of a lesson, givingthe student a break or a digression, repeating a portion of a lesson,changing the difficulty level, types or mix of problems being given, andproviding a tutorial.

All of these adaptations are provided to lessons automatically by thesystem and lesson authors do not need to write them, though they mayconfigure them, by placing limits or parameters on adaptations. Theymight want to do this, for example, in an advanced lesson that uses atool shared by an earlier lesson. In such a lesson, a maximal adaptationmight be undesirable since, if the student needs such a maximaladaptation in order to learn, they might be better served by an earlierlesson.

Intra-Lesson Adaptations

Adaptation within a lesson means a change in any aspect of a lessonbeing delivered, through the overall system or through any tool used todeliver a lesson. A number of such adaptations have been describedpreviously. Additional adaptations include, but are not limited to thefollowing:

The pacing of the lesson can be changed.

Feedback given to the student, including, but not limited to, sounds,volume, visuals, visual effects, and animation, can be changed.

The on-screen layout or visible content of a lesson can be changed.

The size or format of text or other on-screen elements of the lesson canbe changed.

The emphasis of on-screen elements, including, but not limited to,appearance, highlighting, color, appearance, and animation, can bechanged. An alternate version of on-screen element can be substituted.An on-screen element can be added or removed. An on-screen element cancause a different on-screen element to be added or removed. Otheron-screen elements could be faded, disabled, or otherwise modified.

An on-screen annotation of a value, partial value, answer, partialanswer, or information related to a problem or question, possiblysuperimposed over an on-screen element, can be supplied.

An audio annotation of a value, partial value, answer, partial answer,or other information related to a problem, question, or answer can besupplied.

Positive and negative reinforcement given to the student, including, butnot limited to, increasing or decreasing the content, or modifying thetypes of such reinforcement, can be changed.

How answers are disambiguated for students can be changed.

What hints are given and how they are given can be changed.

An alternate response can be provided for a particular on-screenelement. For example, the mouseover effect on a correct answer might beanimated or have a sound effect whereas the mouseover effect on anincorrect answer might be simpler.

With any adaptation, adaptations can be made under lesson control orthey may be continuous adaptations under system control, as describedearlier. For adaptations under lesson control, they may be done inresponse to events that occur or they may be set to happen automaticallyunder certain conditions.

With any adaptation, the adaptation can be made momentarily, for theduration of a problem or longer, or it may be turned and kept on untilit is turned off.

With any adaptation, the adaptation can replace an earlier adaptationinstead of adding to it.

With any adaptation involving on-screen elements, the adaptation itselfcan be sequenced through the elements, as in successive highlighting ofa number of options. For example, counting off a number of items, orsuccessively highlighting or reading the words in a story.

Returning to an example used earlier in this description, there is thecase in early mathematics where most children have an easier timelearning addition than subtraction. The goal is that students need to beable to solve both types of problem without assistance, but moststudents can leverage their knowledge of addition to learn subtractionand this fact may be used to help them learn. As noted previously,clusters of students would be detected with similar outcomes in lessonsabout addition and subtraction. The intra-lesson adaptation makes thisdetection even more relevant because the system will detect clusters ofstudents that perform better when a subtraction problem is given beforean addition problem. The system need not know why this is the case—onlythat it is the case and the system can then leverage this information tohelp those students learn. Further, the system can predict this behaviorin advance of the student even entering such a lesson, by observingwhich other students the student is most similar to.

Note that this predictive adaptation combines with adaptation whichoccurs within a lesson, in response to immediate actions by the student.

Multi-Lesson Adaptations

Another feature of the present invention is planning a complete lessonsequence for a student, in advance, based upon all thecurrently-available lesson information. Such a sequence might berecalculated at particular interval. The spacing between the intervalsmay or may not be fixed, in accordance with different embodiments of theinvention.

Another feature of the present invention is planning one or moreprojected sequences of lessons in order to download more lessons to thestudent in advance. By planning such lesson sequences, the system isable to predict which lessons are most likely to be chosen by thestudent and download the content for those lessons in advance of thembeing chosen, thus minimizing any delays that might occur because ofun-downloaded content.

Student Profile

A feature of the present invention is the maintenance of a “studentprofile” based not upon asking a student what his/her preferred learningstyle is, but upon calculated information about his/her preferred stylein a wide variety of contexts. Advantageously, the system 104 adjuststhis information on an ongoing basis as new information about thestudent is received through the course of progress in lessons. Thus, thesystem 104 is able to deal with learning styles which may be highlycontextual and for which there are no names available. It also does notrequire any knowledge of learning styles on the part of any student oranyone else.

In one embodiment the system 104 system advantageously allowsresearchers to conduct experiments on historical data, to construct andoffer experiments for current students in real time, to conduct suchexperiments without adversely affecting the quality of the educationalmaterials delivered to the students, and to combine the analysis of suchnew experiments with historical data.

By way of example, one such experiment may include what is known as an“A/B test”. In an A/B test one subset (the A group) of the studentsreceives a modified lesson, while the remaining students (the B group,also called the control group) receive the standard or an alternatelesson. Given a large enough student population, multiple A/B tests canbe run simultaneously and the system can analyze data from themindependently. Scientific research favors such A/B tests, but othertypes of experiments are also possible.

FIG. 5 of the drawings shows an exemplary process 500 for running an A/Btest, in accordance with one embodiment. In step 502, a request isreceived for a list of available lessons for a student. In step 504, anumber of substeps are repeated for each of the tests known to thesystem that applies to the student. In step 510, the student's currentwavefront of lessons is interesected with the text matrix. If they donot intersect, the subprocess ends in step 560 and the loop moves on tothe next applicable test. Otherwise, in step 520, the student's profileis examined to see if the student has been assigned to a group for thetest. If not, in step 530, the student is assigned to a test group atrandom and this information is stored in the student profile. In eithercase, processing continues at step 540. In step 540, which group thestudent has been assigned to is inspected. If the student is not in thegroup receiving modified lessons, the subprocess ends in step 560 andthe loop moves on to the next applicable test. Otherwise, in step 550,the modifications or filter that are appropriate for the test areapplied to the lessons database and/or the standards database in such away that any access of them for the subsequent purpose of selecting thelist of lessons for the student causes the modified data to be used.Note that, in essence, an A/B test always involves alternative lessons,but the test may introduce new lessons, the lessons may be modificationsof existing lessons, or the test can be on how, why, or when a lesson isoffered to a student. Next, the process ends in step 360 and the loopmoves on to the next applicable test. When all applicable tests havebeen checked, the loop 504 terminates and processing continues with step506, where the process terminates by proceeding to select the list oflessons for the student.

In alternate embodiments to the one shown in this exemplary process, thestudents could be assigned to test groups in advance, or they could beassigned to test groups by some formula. In addition, there may be morethan two alternatives and it is possible for multiple or allalternatives to introduce modifications to the lessons database and/orthe standards database. In another alternative embodiment, the systemcould dynamically alter the groups of students, or the distribution ofassignment into the groups of students, as information about therelative success of the alternative lessons was learned.

Once an A/B test has been run, no special work is necessary to analyzethe results. Instead, the results of the test are automatically analyzedalong with all other results, using the techniques described above. Thismeans that, if the A/B test was successful (meaning the alternative wasmore successful with some or all students), the system will have learnedit. At this point, the test may be turned off and the alternativelessons or changes to lessons can be introduced into the system. Tostudy the results of an A/B test, a researcher need only examine thedata mining results to see the relative success of the alternativelessons.

Because some jurisdictions may prohibit or regulate such experiments,such as by having laws that require consent and/or parental consent forsuch experimentation, embodiments of the present invention necessarilyintegrate a multi-stage consent model into the system 104.

FIG. 6 of the drawings shows an example of hardware 600 that may be usedto any of the client systems 104 and the server system 106, inaccordance with one embodiment of the invention. The hardware 600typically includes at least one processor 602 coupled to a memory 604.The processor 602 may represent one or more processors (e.g.,microprocessors), and the memory 604 may represent random access memory(RAM) devices comprising a main storage of the hardware 600, as well asany supplemental levels of memory e.g., cache memories, non-volatile orback-up memories (e.g. programmable or flash memories), read-onlymemories, etc. In addition, the memory 604 may be considered to includememory storage physically located elsewhere in the hardware 600, e.g.any cache memory in the processor 602 as well as any storage capacityused as a virtual memory, e.g., as stored on a mass storage device 610.

The hardware 600 also typically receives a number of inputs and outputsfor communicating information externally. For interface with a user oroperator, the hardware 600 may include one or more user input devices606 (e.g., a keyboard, a mouse, heart rate monitor, camera, etc.) and aone or more output devices 608 (e.g., a Liquid Crystal Display (LCD)panel, a sound playback device (speaker), a haptic device, e.g. in theform of a braille output device).

For additional storage, the hardware 600 may also include one or moremass storage devices 610, e.g., a floppy or other removable disk drive,a hard disk drive, a Direct Access Storage Device (DASD), an opticaldrive (e.g. a Compact Disk (CD) drive, a Digital Versatile Disk (DVD)drive, etc.) and/or a tape drive, among others. Furthermore, thehardware 70 may include an interface with one or more networks 612(e.g., a local area network (LAN), a wide area network (WAN), a wirelessnetwork, and/or the Internet among others) to permit the communicationof information with other computers coupled to the networks. It shouldbe appreciated that the hardware 600 typically includes suitable analogand/or digital interfaces between the processor 602 and each of thecomponents 604, 606, 608, and 612 as is well known in the art.

The hardware 600 operates under the control of an operating system 614,and executes various computer software applications, components,programs, objects, modules, etc. to implement the techniques describedabove. Moreover, various applications, components, programs, objects,etc., collectively indicated by reference 616 in FIG. 6, may alsoexecute on one or more processors in another computer coupled to thehardware 600 via a network 612, e.g. in a distributed computingenvironment, whereby the processing required to implement the functionsof a computer program may be allocated to multiple computers over anetwork.

In general, the routines executed to implement the embodiments of theinvention may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions set at various times invarious memory and storage devices in a computer, and that, when readand executed by one or more processors in a computer, cause the computerto perform operations necessary to execute elements involving thevarious aspects of the invention. Moreover, while the invention has beendescribed in the context of fully functioning computers and computersystems, those skilled in the art will appreciate that the variousembodiments of the invention are capable of being distributed as aprogram product in a variety of forms, and that the invention appliesequally regardless of the particular type of computer-readable mediaused to actually effect the distribution. Examples of computer-readablemedia include but are not limited to recordable type media such asvolatile and non-volatile memory devices, floppy and other removabledisks, hard disk drives, optical disks (e.g., Compact Disk Read-OnlyMemory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others,and transmission type media such as digital and analog communicationlinks.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative and not restrictive of the broad invention and thatthis invention is not limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art upon studying thisdisclosure. In an area of technology such as this, where growth is fastand further advancements are not easily foreseen, the disclosedembodiments may be readily modifiable in arrangement and detail asfacilitated by enabling technological advancements without departingfrom the principals of the present disclosure.

1-30. (canceled)
 31. A method for a computing system to adapteducational content for delivery to a student, the method comprising:aggregating, with the computing system, data for each of a plurality ofstudents to form a combined data set, the combined data setcharacterizing interactions of the students with an educational systemwhile responding to questions presented by the educational system, theinteractions of the students with the educational system including stepstaken to resolve the presented questions, the questions associated withan objective within the educational system that was completed by thestudents; automatically fitting, with the computing system, the combineddata set with one or more mathematical models to generate multipleclusters of students having similar skills, each cluster identified by agrouping of students having similar accuracy in responses to thequestions and similar interactions with the educational system whileresponding to the questions, wherein the interactions with theeducational system while responding to the questions include steps takento resolve the questions; receiving, with the computing system,responses to at least some of the questions from a new student that isworking towards the objective within the educational system andmonitoring, with the computing system, interactions of the new studentwith the educational system while responding to the at least some of thequestions; associating, with the computing system, the new student withone of the multiple clusters of students based on the received responsesand the interactions of the new student while responding to thequestions; and adapting and presenting, with the computing system,questions to the new student based on the cluster of students with whichthe new student is associated to provide a customized learningexperience for the new student.
 32. The method of claim 31, furthercomprising predicting an expected accuracy when resolving the questionsfor the new student based on the cluster of students that the newstudent is associated with.
 33. The method of claim 31, wherein adaptingthe questions comprises generating a micro-sequence of lessons topresent to the new student.
 34. The method of claim 33, wherein themicro-sequence of lessons comprises multiple lessons from which thestudent can choose a lesson.
 35. The method of claim 33, whereingenerating the micro-sequence comprises selecting the micro-sequencefrom a set of micro-sequences assigned to the associated cluster, theselection being based on an evaluation of an effectiveness of themicro-sequences in the set.
 36. The method of claim 33, whereingenerating said micro-sequence comprises at least one of altering anexisting micro-sequence and creating a new micro-sequence.
 37. Themethod of claim 31, wherein adapting the questions comprises at leastone of splitting a lesson into multiple lessons, combining multiplelessons into a single lesson, and skipping lessons based on the newstudent's associated cluster.
 38. The method of claim 31, furthercomprising associating the new student with a plurality of the multipleclusters.
 39. The method of claim 31, wherein each cluster is furtheridentified by a learning style associated with the students in thecluster.
 40. The method of claim 31, wherein each of the multipleclusters is based on a situation-dependent differentiation criterion.41. The method of claim 40, wherein the situation-dependentdifferentiation criterion is selected from the group consisting of age,subject matter, gender, school district, and geographic location. 42.The method of claim 31, wherein aggregating the data comprises receivingthe data in the form of an event stream for each student.
 43. The methodof claim 42, wherein the event stream comprises events selected from thegroup consisting of student-generated events, system-generated events,and biometric information pertaining to the student.
 44. The method ofclaim 43, wherein the student-generated events are selected from thegroup consisting of mouse clicks, pen clicks, pointing-device motion,keystrokes, and requests for help by the student.
 45. The method ofclaim 43, wherein the system-generated events comprise a history ofevery screen, problem, image, audio, and video element the educationalsystem generated for the student.
 46. The method of claim 43, whereinthe biometric information is selected from the group consisting of heartrate information, respiration information, eye movement information, andstress level information for the student.
 47. The method of claim 42,wherein the event stream data includes steps taken to resolve eachquestion, a response time for each question, whether a student requesteda hint for each question, and derivatives of the response time for eachquestion.
 48. The method of claim 31, wherein adapting the questions isperformed automatically without human intervention.
 49. The method ofclaim 31, wherein adapting the questions is performed at least in partby a client system that is communicatively coupled to the computingsystem.
 50. The method of claim 31, wherein the interactions with theeducational system while responding to the questions include whetherhints are requested after one or more of the questions are presented.51. The method of claim 31, further comprising predicting an expectedresponse trait when adapting the questions for the new student based onthe cluster of students with which the new student is associated. 52.The method of claim 31, wherein adapting the questions delivered to thenew student comprises adapting a type of question and adapting a mannerof delivery of the question.
 53. One or more computing systems foradapting educational content for delivery to a student, the one or morecomputing systems comprising: one or more computer-readable storagemediums storing computer-executable instructions for controlling the oneor more computing systems to: aggregate data for each of a plurality ofstudents to form a combined data set, the combined data setcharacterizing interactions of the students with an educational systemwhile responding to questions presented by the educational system, theinteractions of the students with the educational system including stepstaken to resolve the presented questions, the questions associated withan objective within the educational system that was completed by thestudents; automatically fit the combined data set with one or moremathematical models to generate multiple clusters of students havingsimilar skills, each cluster identified by a grouping of students havingsimilar accuracy in responses to the questions and similar interactionswith the educational system while responding to the questions, whereinthe interactions with the educational system while responding to thequestions include steps taken to resolve the questions; receiveresponses to at least some of the questions from a new student that isworking towards the objective within the educational system and monitorinteractions of the new student with the educational system whileresponding to the at least some of the questions; associate the newstudent with one of the multiple clusters of students based on thereceived responses and the interactions of the new student whileresponding to the questions; and adapt and present questions to the newstudent based on the cluster of students with which the new student isassociated to provide a customized learning experience for the newstudent; and one or more processors for executing thecomputer-executable instructions stored in the one or morecomputer-readable storage mediums.
 54. The one or more computing systemsof claim 53, wherein the computer-executable instructions furthercontrol the one or more computing systems to predict an expectedaccuracy when resolving the questions for the new student based on thecluster of students that the new student is associated with.
 55. The oneor more computing systems of claim 53, wherein adapting the questionscomprises generating a micro-sequence of lessons to present to the newstudent.
 56. The one or more computing systems of claim 55, wherein themicro-sequence of lessons comprises multiple lessons from which thestudent can choose a lesson.
 57. The one or more computing systems ofclaim 55, wherein generating said micro-sequence comprises selecting themicro-sequence from a set of micro-sequences assigned to the associatedcluster, the selection being based on an evaluation of an effectivenessof the micro-sequences in the set.
 58. The one or more computing systemsof claim 55, wherein generating the micro-sequence comprises at leastone of altering an existing micro-sequence and creating a newmicro-sequence.
 59. The one or more computing systems of claim 53,wherein adapting the questions comprises at least one of splitting alesson into multiple lessons, combining multiple lessons into a singlelesson, and skipping lessons based on the new student's associatedcluster.
 60. The one or more computing systems of claim 53, wherein thecomputer-executable instructions further control the one or morecomputing systems to associate the new student with a plurality of themultiple clusters.
 61. The one or more computing systems of claim 53,wherein each cluster corresponds to a learning style associated with thestudents in the cluster.
 62. The one or more computing systems of claim53, wherein each of the multiple clusters is based on asituation-dependent differentiation criterion.